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ABSTRACT 



This paper describes the processes that one school district, 
the Visalia Unified School District (California) , went through in 
establishing a standards-based assessment and accountability system using 
multiple methods. First the district’s standards-based assessment and 
accountability system and the methods used to establish it are described, and 
the approaches the district explored in setting the performance standards are 
outlined. After exploring publisher-developed standards, the district settled 
on setting the performance standards separately for each grade as recommended 
by the California Department of Education. The multiple measures used by the 
school district include norm- referenced and performance-based assessments 
from the TerraNova Multiple Assessments published by CTB/McGraw-Hill and 
their Spanish equivalents. The second part of the paper presents achievement 
results along with the adequate yearly progress target and disaggregated 
achievement data. The paper concludes with a discussion of implications for 
district and state policy for implementing the standards-based assessment and 
accountability system. (Contains 16 tables and 22 figures.) (SLD) 
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Introduction 



The reauthorized Elementary and Secondary School Act (ESEA) called Improving 
America’s School Act (IASA) requires states to set high academic content and performance 
standards for all students. One major element of this law is the “standards-based” accountability 
for local school districts and schools receiving Title I (previously Chapter I) funds. To 
implement the “standards-based” accountability, the California Department of Education (CDE) 
developed a statewide standards-based accountability system for local school districts and 
schools in California. According to these guidelines, school districts are required to establish a 
standards-based assessment and accountability system for all schools, including non-Title I 
schools. Districts and schools are required to report student achievement data based on the 
district adopted or developed performance standards, following the guidelines set by the CDE. 

Districts and schools in California are faced with challenges in developing a standards- 
based assessment and accountability system that monitors student achievement and progress, and 
provide information that will help schools and teachers enhance student achievement through 
effective teaching and learning practices. The California Department of Education has set a goal 
for each school district and for each school to have at least 90 percent of students performing at 
or above grade-level standards in no more than ten years (i.e., in 2006-07). 

To reach this goal, schools are expected to make adequate yearly progress each year, with 
1996-97 as base year. Adequate yearly progress is the yearly increase in percent of students 
meeting or exceeding grade-level standards that will be needed to reach the goal of 90 percent in 
ten years. In 1996-97 and 1997-98, Title I schools will be candidates for Program Improvement 
if student performance data show that fewer than 40% of the students are meeting or exceeding 
grade-level standards. 

This paper describes the processes that one school district went through in establishing a 
standards-based assessment and accountability system using multiple measures. First, the 
district’s standards-based assessment and accountability system as well as the methods and 
procedures used to establish the system are described, followed by the several approaches that the 
district explored in setting the performance standards. The second part of the paper presents 
achievement results along with the adequate yearly progress target and disaggregated 
achievement data. The last section of the paper concludes with the implications for district and 
state level policy for implementing the standards-based assessment and accountability system. 
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Visalia Unified Assessment and Accountability System 

The district’s standards-based Assessment and Accountability System is presented in this 
section. It describes the measures the district employed in 1996-97 to identify the level of 
performance for individual students in reading/language arts and mathematics m grades K-12. It 
also describes the methods and procedures used to obtain a single standards-based performance 
level for individual students, including 

(a) the weight given to each measure in combining the results, and 

(b) the criteria/methods used to establish grade-level standards. 



Description of Measures 

Visalia Unified has a multi-year plan for developing a comprehensive assessment system 
that consists of multiple measures for assessing student achievement. The district believes that a 
comprehensive assessment system is an integral part of a high quality educational program or 
students. The goal of the district’s assessment system is to: 

► improve the quality of student learning 

► provide accurate information about student achievement in all subjects and skill 
areas that will be used by teachers, schools, the district and the community to 
improve instructional programs for all students 

► serve as a means by which both the staff and community can measure the 
progress of all students toward achieving the district’s “List of Agreements. 



The guiding principles of the district’s assessment system are to: 

► integrate teaching, learning and assessment into a system that creates and upholds 
consistently high standards for student performance 

► provide students with an understanding of the purposes and benefits of a quality 

assessment system 

► measure both specific knowledge and critical thinking 

► incorporate more than one type of assessment to fairly evaluate student progress 

► be practical to administer and provide accurate, useful and timely data 

► be responsive to all students regardless of experience, background and abilities 

► prepare teachers for an active role in analysis of student work to improve teaching 

and learning 

► be reported to stakeholders in an understandable format 
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Implementing Local Standards-Based Assessment and Accountability System 
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Introduction 

The reauthorized Elementary and Secondary School Act (ESEA) called Improving 
America’s School Act (IAS A) requires states to set high academic content and performance 
standards for all students. One major element of this law is the “standards-based accounta 1 lty 
for local school districts and schools receiving Title I (previously Chapter I) funds. To 
implement the “standards-based” accountability, the California Department of Education (CDE) 
developed a statewide standards-based accountability system for local school districts and 
schools in California. According to these guidelines, school districts are required to estabhs a 
standards-based assessment and accountability system for all schools, including non-Title I 
schools. Districts and schools are required to report student achievement data based on the 
district adopted or developed performance standards, following the guidelines set by the CDE. 

Districts and schools in California are faced with challenges in developmg a standards- 
based assessment and accountability system that monitors student achievement and progress and 
provide information that will help schools and teachers enhance student achievement through 
effective teaching and learning practices. The California Department of Education has set a goal 
for each school district and for each school to have at least 90 percent of students performing at 
or above grade-level standards in no more than ten years (i.e., in 2006-07). 

To reach this goal, schools are expected to make adequate yearly progress each year, 
with 1996-97 as base year. Adequate yearly progress is the yearly increase in percent of students 
meeting or exceeding grade-level standards that will be needed to reach the goal of 90 percent m 
ten years In 1996-97 and 1997-98, Title I schools will be candidates for Program Improvement 
if student performance data show that fewer than 40% of the students are meeting or exceeding 

grade-level standards. 

This paper describes the processes that one school district went through in establishing a 
standards-based assessment and accountability system using multiple measures. First, the 
district’s standards-based assessment and accountability system as well as the methods and 
procedures used to establish the system are described, followed by the several approaches that 
the district explored in setting the performance standards. The second part of the paper presents 
achievement results along with the adequate yearly progress target and disaggregated 
achievement data. The last section of the paper concludes with the implications for district and 
state level policy for implementing the standards-based assessment and accountability system. 
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Visalia Unified Assessment and Accountability System 

The district’s standards-based Assessment and Accountability System is presented in this 
section. It describes the measures the district employed in 1996-97 to identify the level of 
performance for individual students in reading/language arts and mathematics in grades K-12. It 
also describes the methods and procedures used to obtain a single standards-based performance 
level for individual students, including 

(a) the weight given to each measure in combining the results, and 

(b) the criteria/methods used to establish grade-level standards. 

Description of Measures 

Visalia Unified has a multi-year plan for developing a comprehensive assessment system 
that consists of multiple measures for assessing student achievement. The district believes that a 
comprehensive assessment system is an integral part of a high quality educational program for 
students. The goal of the district’s assessment system is to: 

► improve the quality of student learning 

► provide accurate information about student achievement in all subjects and skill 
areas that will be used by teachers, schools, the district and the community to 
improve instructional programs for all students 

► serve as a means by which both the staff and community can measure the 
progress of all students toward achieving the district’s “List of Agreements.” 

The guiding principles of the district’s assessment system are to: 

► integrate teaching, learning and assessment into a system that creates and upholds 
consistently high standards for student performance 

► provide students with an understanding of the purposes and benefits of a quality 
assessment system 

► measure both specific knowledge and critical thinking 

► incorporate more than one type of assessment to fairly evaluate student progress 

► be practical to administer and provide accurate, useful and timely data 

► be responsive to all students regardless of experience, background and abilities 

► prepare teachers for an active role in analysis of student work to improve teaching 
and learning 

► be reported to stakeholders in an understandable format 
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The district is in its second year of developing a comprehensive assessment system. The 
next phase of the district's assessment system will focus on developing assessment too s in 
mathematics for grades K and 1 and performance-based assessments in at least one grade rom 
each grade span: grades 3-5, 6-9, and 10-12 in reading/language arts and mathematics. 

In 1996-97 school year, the district's assessment system included multiple measures that 
consists of both norm-referenced test and performance-based assessments. For grades K-l 
reading/language arts, the district administered classroom embedded Early Literacy 
Observational Surveys from the Early Literacy Project to all K-2 students m the district, 
including students with disabilities and limited-English proficient (LEP) students. For 
Kindergarten, three measures were administered in the Spring of 1997: Letter Identification, 
Hearing and Recording Sounds in Words and Writing Vocabulary. For first grade. Text Reading 
(Running Record) was administered. The Spanish equivalent of these measures was 
administered to our Spanish speaking LEP students. The results from the Text Reading along 
with student grades are used to report student achievement m readmg/language arts for first 
grade In the 1997-98 school year, a Writing Test in addition to Text Reading will be 
administered to all first graders. For grades K-l in math, the district did not have any valid and 
reliable measures to administer in the 1996-97 school year. Student grades were used to report 

the K-l student achievement in mathematics. The district will focus on developing K- 1 math 

assessments in the 1997-98 school year. 

For grades 2-10 reading/language arts and mathematics, the TerraNova and Sup'era 
(Spanish Equivalent of TerraNova) Multiple Assessments, published by CTB/McGraw-Hill, 
were administered to all students in grades 2-10 in the Spring of 1997, including special 
education and limited-English proficient (LEP) students in the district. Accommodauons were 
made for testing of students with disabilities. The Supera was administered to our Spanish 
speaking LEP students. The TerraNova and Supera Multiple Assessments consist of test items 
that included both selected-response (SR) items and constructed-response (CR) items. Tables 1 
and 2 display the percent of SR items and CR items included in the multiple assessments. 

Scores on all pieces of these items are combined to yield one single score in readmg/language 
arts and mathematics. 

Student academic grade on Junior English for grade 1 1 and Writing Proficiency Exam 
for grade 12 are used to report the student achievement in reading/language arts. For grades 
11-12 mathematics, student academic grades on both algebra and geometry are used for 
reporting student achievement. Tables 1 and 2 show a matrix of the district's assessment and 
accountability system and the percent of the weight assigned to each measure to determine the 
percent of students meeting or exceeding the grade level standard in reading/language arts and 

mathematics, respectively. 
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Table I: Matrix of 1996-97 Visalia Unified Assessment System and Weight Assigned to Each 
Measure for Reading/Language Arts (percent) 



Measures 


Reading/Language Arts 


Letter 

Identification 


Hearing & 
Recording 
Sounds in 
Words 


Writing 

Vocabulary 


Text 

Reading 


Grade 

on 

Report 

Cards 


TerraNova & 
Supera Multiple 
Assessments* 


Grade on 

Junior 

English** 


Writing 

Proficiency 

Exam*** 


SR 


CR 


Grade K 


Slilfl 


















Grade 1 








Igp^gg 












Grade 2 




















Grade 3 




















Grade 4 




















Grade 5 














SIIS1 






Grade 6 












§§J6$|g 








Grade 7 




















Grade 8 














liltf 






Grade 9 














if. 






Grade 10 












jfpjppi 


IIS 






Grade 11 




















Grade 12 





















* TeiraNova and Sup era Multiple Assessments consist of test items that include both selected-response (SR) items and 
constructed-response (CR) items. The percent of SR items and CR items in the test is indicated in the table. Under item 
pattern scoring, weight of each item is a function of item discrimination and variability of student responses. Scores on 
all pieces of these items are combined to yield one single score in reading/language arts and mathematics. 

** Included students who passed Junior English by the end of their 1 1th grade. 

** ^Included students who passed the Writing Proficiency Exam by the end of their 12th grade. 
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Table 2: Matrix of 1996-97 Visalia Unified Assessment System and Weight Assigned to Each 
Measure for Mathematics (percent) 





Mathematics 


Measures 


Grade on 

Report 

Cards 


TerraNova & 
Supera Multiple 
Assessments* 


Grade on 
Algebra & 
Geometry** 




SR 


CR 


Grade K 


III 100% || 








Grade 1 










Grade 2 




wmm 


BS$iS# 




Grade 3 






issiiit 




Grade 4 




Sfiiili 






Grade 5 




fj||||§t 






Grade 6 




funs 


SSSlSf 




Grade 7 




ssfifti 


wm 




Grade 8 




iSm 


ws&xi 




Grade 9 




IS8SS1 


iii^n 




Grade 10 










Grade 11 










Grade 12 








ISHSlM 



* TeiraNova and Supera Multiple Assessments consist of test items that include both selected-response (SR) items and 
constructed-response (CR) items. The percent of SR items and CR hems in the test is indicated in the table. Under item 
pattern scoring, weight of each item is a function of hem discrimination and variability of student responses. Scores on 
all pieces of these items are combined to yield one single score in reading/language arts and mathematics. 

♦♦Included students who passed both algebra and geometry by the end of 11th grade for grade 11 and by the end of 12th 

grade for grade 12. 



Description of Methods Used for Setting Visalia Unified Performance Standards 

Three levels of performance standards were established for grades K- 1 reading/language 

arts: 

1 Advanced (above grade-level standard), 

2. Proficient (at grade-level standard) and 

5 
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3. 



Partially Proficient (below grade-level standard). 



In setting the grade-level standards, scores from the Early Literacy measures were used. 
Tables 3 and 4 show the score cut points for each performance level - at, above and below 
standards - and the percent of weight assigned to each measure for Kindergarten and first grade, 
respectively. For example, Letter Identification score 36 or above represents at or above 
grade-level standards for Kindergarten. Writing Vocabulary score 7 or below represents below 
grade-level standards. In judging student performance, Letter Identification measure was 
weighted 40%, Hearing and Recording Sounds in Words 35% and Writing Vocabulary 25% for 
Kindergarten. For first grade. Text reading was weighted 70% and grade was weighted 30%. 

Visalia Unified has an Early Literacy Training Team that consists of six teachers. These 
teachers provide inservice to all K-2 teachers in the district to implement the Early Literacy 
Program. The performance standards were developed by the committee that consists of the 
Early Literacy Training Team and district staff from the elementary education, compensatory 
education and evaluation and assessment offices. Research studies from the Early Literacy 
Program, Reading Recovery Program and our students data from the Observational Surveys 
were examined along with the published statistical information in setting the performance 
standards for grades K-2 reading/language arts. 

K- .1 Reading/Language Arts Assessment Performance Standards 

Table 3: Kindergarten Reading/Language Arts Assessment Performance Standards 



Measures 


Performance Standards 


Scores 


i. 


Letter Identification 


3 Advanced 


Above standards 


50+ 


(weight: 40%) 


2 Proficient 


At standards 


36-49 


1 Partially Proficient 


Below standards 


0-35 


2. 


Hearing & Recording 
Sounds in Words 


3 Advanced 


Above Standards 


32+ 


(weight: 35%) 


2 Proficient 


At standards 


22-31 


1 Partially Proficient 


Below standards 


0-21 


3. 


Writing Vocabulary 


3 Advanced 


Above Standards 


13+ 


(weight: 25%) 


2 Proficient 


At standards 


8-12 


1 Partially Proficient 


Below standards 


0-7 
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Table 4: First Grade Reading/Language Arts Assessment Performance Standards 



Measures 


Performance Standards 


Scores 


i. 


Text Reading 


3 Advanced 


Above standards 


18+ 


(weight: 70%) 


2 Proficient 


At standards 


14- 16 


1 Partially Proficient 


Below standards 


12 or below 


2. 


Report Cards 


3 Advanced 


Above Standards 


“Excellent” 


(weight: 30%) 


2 Proficient 


At standards 1 


“Satisfactory” 


1 Partially Proficient 


Below standards 


“Progressing, 
below grade 
level (P)” or 
“Needs 
Improvement 
(NT 



For kindergarten mathematics, two performance levels were established: Proficient (at 
grade-level standard) and Partially Proficient (below grade-level standard), using information 
from the student report cards. "Satisfactory growth" represents at grade-level standard and 
"progress but needs to continued experience" represents below grade-level standard (See Table 
5). For first grade mathematics, three levels of performance standards were established. 

Student grade from report cards was also used in setting these standards. "Excellent represents 
above grade-level standard and "Satisfactory" represents at grade-level standard. "P" or "N" 
represents below grade-level standard (See Table 6). 



1C- 1 Math Assessment Performanc e Standards 

Table 5: Kindergarten Mathematics Performance Standards 



Measures 


Performance Standards 


Scores 


i. 


Report Cards 


2 Proficient 


At standards 


“Satisfactory growth (✓)” 


1 Partially Proficient 


Below standards 


“Progressing, but needs 
continued experiences (P)” 



Table 6: First Grade Mathematics Performance Standards 



Measures 


Performance Standards 


Scores 


i. 


Report Cards 


3 Advanced 


Above Standards 


“Excellent” 


2 Proficient 


At standards 


“Satisfactory” 


1 Partially Proficient 


Below standards 


“Progressing, below grade level 
(P)” or “Needs Improvement (N)” 



Grades 2-10 Reading/Language Arts Performance Standards 

Described below are the methods and procedures used to establish the district's 
performance standards in grades 2-10 for reading/language arts and mathematics based on the 
guidelines set by the California State Department of Education's Statewide Accountability 
System. The TerraNova and Supera Multiple Assessments were administered to all students in 
grades 2-10. Four levels of performance standards were established: 

1. Advanced (exceed or above grade-level standards), 

2. Proficient (meet or at grade-level standards), 

3. Partially Proficient (below grade-level standards) and 

4. Below Partially Proficient (below grade-level standards). 

A National Percentile (NP) score is used in setting the district's performance standards in 
grades 2-10. The state requires that the 50th National Percentile or above should be used in 
judging student performance for meeting the grade-level standards. In setting the proficient 
level performance, the 50th percentile is used as- a cut point. The 76th percentile, which 
represents the top quarter of percentile, is used as a cut point in setting the advanced level 
performance. Table 7 shows the NP cut points for each performance level across grade levels 
2 - 10 . 



Table 7: Grades 2-1 0 Reading/Language Arts and Mathematics Performance Standards 



Performance Level 


National Percentile 


Advanced (Above or Exceed Grade-Level Standards) 


76 and above 


Proficient (At or Meet Grade-Level Standards) 


50 - 75 


Partially Proficient (PP) (Below Grade-Level Standards) 


35-49 


Below PP (Below Grade-Level Standards) 


34 and below 
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Partially proficient (PP) level is established to show the percent of students making 
progress from partially proficient to proficient each year. The purpose of this information is to 
assist schools in monitoring their instruction and programs to better meet the needs of these 
students. A National Percentile of 35, previously the Title I identification cut point, is used in 
setting the partially proficient level. 

Several approaches were explored and reviewed internally in setting the performance 
standards. The CTB/McGraw-Hill's performance standards were examined. Alternative 
approaches to the publisher's performance standards were also explored. A study session on 
setting performance standards was held with the school board and the public. Advantages and 
disadvantages along with the technical qualities on each approach were discussed and examined 
during this study session. The district adopted the performance standards that will result m 
more valid and consistent information as well as the most meaningful to the district and local 
schools, following the recommendations from the California State Department of Education. 



Grades 11-12 Reading/Language Arts and Math Performance Standards 

Student academic grade was converted into a standard score: 3, 2 or 1. Table 8 shows 
each grade's corresponding standard score. Grade "A" represents above grade-level standard. 
Grades "B" or "C" represents at grade-level standard. The percent of students with standard 
scores 2 or 3 is reported as meeting or exceeding grade-level standards. 



Table 8: Grades 11-12 Reading/Language Arts and Mathematics Performance Standards 



Grade 


Performance Standards 


Standard Score 


“A” 


Above Standard 


3 (pass) 


“B” or “C” 


At Standard 


2 (pass) 


“D” or “F” 


Below Standard 


1 (fail) 



The methods and procedures used in determining the percent of students at or above 
grade-level standards in reading/language arts and mathematics are as follows. Student grades 
in Junior English for grade 1 1 and the Writing Proficiency Exam (WPE) for grade 12 were used 
in determining the percent of students meeting or exceeding grade-level standards m reading/ 
language arts. Student grades on both algebra and geometry courses was used in determining 
the percent of students meeting or exceeding grade-level standards in mathematics for grades 1 1 

and 12. 



All of the students who were enrolled and passed Junior English by the end of grade 1 1 
(includes grades 9 or 10) and WPE by the end of grade 12 (includes grades 9, 10, or 1 1) were 
included in the calculation. All of the students who were enrolled and passed both algebra and 
geometry by the end of grade 1 1 (includes grades 9 or 10) and by the end of grade 12 (includes 
grades 9, 10, or 1 1) were included in calculating the percent of students meeting or exceeding 
grade level standards. The chart below illustrates the formulas for grades 1 1 & 12 calculation. 

GRADE 11: 



Percent of students at/above grade-level standards = Number of students vassedin grades, 9. 10. or 11 

Total number of grade 11 student population 



GRADE 12 



Percent of students at/above grade-level standards = grades^ 9. 10. 77. or 1 2 

Total number of grade 12 student population 



Procedures for combining grades on both algebra and geometry to determine the 
grade-level standard are illustrated in Table 9. As Table 9 illustrates, the grade on both 
algebra and geometry from each semester were converted into a standard score. These 
standard scores were then added together. The total score was divided by the number of 
semesters. The final score 2.5 was rounded up to 3. The score 3 represents above grade level 
standard. 

Table 9: Illustration of Combining Student Grade to Determine Grade-Level Standards 



Course 


Semester 


Student Grade 


Standard 

Score 


Final Score 


Calculation 


Algebra 


Semester 1 


"A" 


3 








Semester 2 


"C" 


2 






Geometry 


Semester 1 


"B" 


2 








Semester 2 


"A" 


3 






Sum 


10 


3 (pass) 


(3+2+2+3)/4=2.5 
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ELD Standards 



ELD standards are set by the number of years that a student has received ELD 
instruction using the Student Oral Language Observation Matrix (SOLOM) test. Table 10 
shows SOLOM score cut points by the number of years of ELD instruction a student received. 
As can be seen in Table 10, for the first year of ELD instruction, a SOLOM score 8 or above 
represents that a student is meeting the standard for developing his/her language proficiency in 
English. For a student to be judged as fluent in English using SOLOM, he/she must score 23 

or above. 



Table 1 0: ELD Standards by Number of Years of Instruction . 



Year in ELD Instruction 


SOLOM Scores 


1st 


8+ 


2nd 


13+ 


3rd 


18+ 


4th and 5th 


23+ 



Approaches To Setting Performance Standards 

This section of the paper describes the methods that the district studied in setting the 
performance standards using the norm-referenced standardized achievement test. The district 
administered the TerraNova and Supera (Spanish equivalent of TerraNova) Multiple 
Assessments, published by CTB/McGraw-Hill, to students in grades 2-10 in the Spring of 
1997. In setting the district’s performance standards, several approaches were explored and 
reviewed internally. First, the publisher’s performance standards setting approach was 
examined. Second, alternative approaches to the publisher's performance standards were 
explored. A study session on setting performance standards was held with the school board 
and the public. Advantages and disadvantages along with the technical qualities of each 
approach were discussed and examined during this study session. Below describes each of the 
four approaches that the district examined in setting the performance standards. 



First Approach: Use publisher developed, performance level standards 



Table 11: Publisher j Recommended Grade Level Cut Scores in National Percentiles (NP) 



Grade 


Proficient 


Advanced 


Reading 


Language 


Math 


Reading 


Language 


Math 


2 


59 


61 


60 


89 


90 


90 


3 


86 


87 


95 


97 


97 


99 


4 


76 


80 


85 


93 


95 


97 


5 


67 


68 


70 


91 


91 


90 


6 


82 


81 


88 


96 


95 


97 


7 


75 


76 


79 


94 


93 


94 


8 


67 


69 


67 


90 


91 


90 


9 


89 


90 


91 


98 


98 


99 


10 


85 


87 


88 


97 


98 


98 



Within the context of California Accountability System, which recommends that 
content and performance standards should be developed on a grade by grade basis, this first 
approach does not provide valid information by grade level. Rather than developing 
performance standards for each grade separately, the publisher chose to develop standards by 
grade spans: grades 1-2, grades 3-5, grades 6-8 and grades 9-12. For example, the grade span 
3-5, the same standards apply for grades 3, 4 and 5. This means, by design, the number of 
students meeting the proficient and advanced level will be smaller for the fourth grade and still 
much smaller for the third grade compared to students in the fifth grade. As can be seen in 
Table 1 1, a NP cut score in reading for third grade is much higher (86 percentile) than that of 
the fourth grade (76 percentile) and fifth grade (67 percentile). This is likely to give a 
misleading picture of student achievement because it will appear that fourth and third grade 
students are not performing as well as the fifth grade students. 

Figures 1-3 shows the percent of the students in the norm group that performed at or 
above standards in reading, language and mathematics based on the CTB’s performance 
standards. This approach was not recommended on technical ground. 



Insert Figures 1-3 Here 
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Figures 1-3: Percent of Students in the Norm Group At (Proficient) or Above (Advanced) Standards 
Based on the CTB's Standards 

Figure 2: 







Second Approach: Use publisher developed, performance standards and interpolate National 
Percentiles, (NP) using, benchmark grade level (grades 2, 5. 8. and 12} NP score 



Table 12: Interpolated Grade Level Cut Scores Based on Publisher 's Benchmark Grade 

Levels (2, 5, 8 and 12) In National Percentiles (NP) 



Grade 


Proficient 


Advanced 


Reading 


Language 


Math 


Reading 


Language 


Math 


2 


59 


61 


60 


89 


90 


90 


3 


62 


63 


63 


90 


90 


90 


4 


65 


65 


66 


90 


90 


90 


5 


67 


68 


70 


91 


91 


90 


6 


67 


68 


69 


90 


91 


90 


7 


67 


68 


68 


90 


91 


90 


8 


67 


69 


67 


90 


91 


90 


9 


68 


70 


69 


90 


91 


91 


10 


69 


71 


70 


91 


91 


91 



This approach uses the publisher’s performance standards and interpolates National 
Percentile (NP) cut scores using the benchmark grade level - grades 2, 5, 8, and 12. For 
example, a second grade NP cut score for proficient level in reading is 59. A fifth grade NP 
cut score for proficient level in reading is 67. The NP score difference between grades 2 and 5 
in reading is 8 points. This score difference, 8 points, is divided by 3, the number of grades 
between grades 2 and 5. This number, 2.67 (8 divided by 3), is evenly distributed between 
grades 2 through 5. Using this approach, a NP cut point for grades 2, 3, 4 and 5 for proficient 
level in reading is 59, 62, 65 and 67, respectively. 

Performance standards for the second approach are set on a grade by grade basis, 
unlike the first approach. However, these standards are external to the district and the state. 
They are not developed for California specifically. Thus, these standards are less meaningful 
to the district and local schools. Rather, they are more comparable to the national standards 
such as National Assessment of Educational Progress (NAEP), which are very stringent 
standards. 
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7 jMJtmSChL I Ivhnrhmark erode level (trades ? 5 S. I?) National Percentiles MP Lto 
sej the performance standar d s {or other grades. 

Table 13: Benchmark Grade Level (2, 5, 8 and 12) National Percentiles Applied To Each 



Grade Level Within Grade Span: 1-2, 3-5, 6-8 9-12 



Kjraue mvei rvtimn 
Grade 


Proficient 


Advanced 


Reading 1 Language | Math 


Reading 1 Language | 


Math 


2 


59 


61 


60 


89 


90 


90 


3 


67. 


68 


70 


91 


91 


90 


4 


67 


68 


70 


91 


91 


90 


5 


67 


68 


70 


91 


91 


90 


6 


67 


69 


67 


90 


91 


90 


7 


67 


69 


67 


90 


91 


90 


8 


67 1 


69 


67 


90^ 


91 


90 


9 


71 


72 


73 


92 


| 91 1 


93 


10 


71 


72 


73 


92 


91 


93 



The third approach to setting the standards examined the publisher s benchmark grade 
level NP cut scores and used the benchmark grade level National Percentiles to set the 
standards for other grade levels within each grade span: grades 1-2, grades 3-5, grades 6-8 and 
grades 9-12. For example, a NP score of 67 for grade 3 was applied to also grades 4 and 5. 
This approach was not recommended since it is based upon many assumptions which may or 
may not be valid because the same standards are used across the grade span. 

Fourth Approach! Set local standards us ing. National Eercenlik (HEX score 



Table 14: Local Performance Standards National^ Percentile^ Cut Scores^ 





Proficient 


Advanced 


Grade 


Reading 1 


Language | 


Math 


Reading fl 


Language. | 


Math 


2 


50 


50 


50 


76 


76 


76 


3 


50 


50 


50 


76 


76 


76 


4 


50 


50 


50 


76 


76 


76 


5 


50 


50 


50 


76 


76 


76 


6 


50 


50 


50 


76 


76 


76 


7 


50 


50 


50 


76 


76 


76 


8 


50 


50 1 


50 


76 


I 76 1 


76 


9 


50 


50 


50 


76 


76 


76 


10 


50 


50 


50 


76 


76 


76 
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In the fourth and last approach to setting the standards, the 50th percentile is used as a 
cut point in setting the proficient level. The 76th percentile, which represents the top quarter 
of percentile, is used as a cut point in setting the advanced level performance. The fourth 
approach is consistent with the guidelines set by the California Department of Education 
(CDE). The state requires that the 50th National Percentile or above should be used in judging 
student performance for meeting the grade-level standards. The CDE also recommends that 
the standards should be developed on a grade by grade basis. In this approach, standards are 
set consistent across grade levels. This procedure will result in more consistent estimates that 
will be comparable to other districts because most districts are likely to set their standards at 
the 50th percentile level following the CDE guidelines. 

An in-depth study session was conducted with the local school board and the public 
about all approaches to setting the performance standards. Advantages and disadvantages 
along with the technical qualities pertaining to each approach were examined. The second and 
the fourth approach was examined and discussed as possible standards for the district. One of 
the difficulties for adopting the performance standards developed by the test publisher was that 
those standards were not particularly relevant to the state and district. They were not 
developed for California specifically, rather these standards were more closely aligned with the 
standards such as NAEP. Therefore, it was concluded that they were less meaningful to the 
local district and schools. 

Finally, the district took the approach to set the performance standards for each grade 
independently as recommended by the CDE. It was also concluded that the local approach 
will result in more valid and consistent information as well as the most meaningful to the 
district and schools, following the recommendations from the CDE. The district adopted the 
fourth approach in setting the performance standards. Figures 4-6, 7-9, and 10-12 present the 
district achievement data pertaining to approaches one, two and four respectively. 



Insert Figures 4-6, 7-9 and 10-12 Here 
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Achievement Results and Disaggregated Data 

This part of the paper summarizes student achievement results for the district and for 
each school in the district. Achievement results are reported by the percent of students at or 
above grade-level standards in reading/language arts and mathematics. Achievement results 
are based on the students who were in the school for a full academic year, that is, students who 
were enrolled in the school since the first month of the school year. Student achievement 
results are disaggregated by the following categories: 

1 . Title I students 

2. Migrant students 

3 . Language proficiency 

a. Limited-English Proficient (LEP) students 

b. Fluent-English Proficient (FEP) students 

c. English Only (EO) students 

d. Redesignated-Fluent-English Proficient (R-FEP) students 

4. Special education students 

5. Gifted and Talented (GATE) students 

6. Ethnicity 

7. Educationally Disadvantaged Students - free/reduced lunch program vs. paid 

lunch program 

8. Gender 

9. English Language Development (ELD) 

Disaggregated achievement results are based on the students who were in the school for 
a full academic year, except for the migrant students. For migrant students, achievement 
results are based on all migrant students assessed, regardless of length of time in the school. 

The districtwide accountability summary by the percent of students at or above the 
grade-level standards combining content areas (see Table 15 and Figure 13) and by each 
content area, reading/language arts and mathematics are presented (see Figures 14-15). Table 
16 shows the percent of the students at or above the grade level standards for each school in 
the district. An adequate yearly progress growth target for each school in 1997-98 year to 
reach the goal of 90 percent of students meeting or exceeding the grade-level standards in ten 
years is also presented (see Figures 16 & 17). Figures 18-37 present the disaggregated data by 
programs and subgroups summarized by content areas, reading, language and mathematics. 



Insert Table 15 and Figures 13-15 Here 
Insert Table 16 Here 
Insert Figures 16 and 17 Here 
Insert Figures 18-37 Here 
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Table 16: Visalia Unified School District Accountability Summary Report By School 



School Site;£f|j 


^l?ercxM-6^fad^ts At ora 
W^A^ve;:Stomkirds : (%)£$& 


Meet 40 % A cajuntabi li ty'i 


^£^0997-98 (%j 


CO 


61 


✓ 


3.2 


cw \ 


52 


✓ 


4.2 


CR * 


26 




7.1 


EC 


48 


✓ 


4.7 


FV * 


33 




6.3 


GK 


47 


✓ 


4.8 


GO * \ 


36 




6.0 


m * 


47 


✓ 


4.8 


HO * 


15 




8.3 


i?T/ 1 


56 


✓ 


3.8 


JK* j 


33 




6.3 


LW 


50 


✓ 


4.4 


MK * 


39 




5.7 


A/K 


50 


✓ 


4.4 


PI 


62 


✓ 


3.1 


RB" 


0 




10.0 


RO 


64 


✓ 


2.9 


UN* 


40 


✓ 


5.6 


VB 


53 


✓ 


4.1 


WA * 


33 




6.3 


WG 


42 


✓ 


5.3 


DIV 


37 




5.9 


GA 


33 




6.3 


U 


51 


✓ 


4.3 


VO 


39 




5.7 


GW 


62 


✓ 


3.1 


MW 


66 


✓ 


2.7 


RW 


60 


✓ 


3.3 


SQ 


35 




6.1 


PW *** 


10 




8.9 



* Title I Schools; **100 percent special education school; ***Altemative school 
Accountability summary is across grade levels and subject areas, reading/language arts and mathematics. 
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Standards-Based Student Achievement Results 
Districtwide Summary by Disaggregated Categories 
Percent of Students At or Above Standards 
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Standards-Based Student Achievement Results 
Districtwide Summary by Ethnicity 
Pgrro-nt of Students At or A bove Standards 

Figure 27: 






Standards-Based Student Achievement Results 
Districtwide Summary by Disaggregated Categories 
Percent of Students At or Above Standards 
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Implications for District and State Level Policy 

Accountability is a very sensitive and important issue for the schools, districts, and the 
local school boards. In spite of the good intentions on the part of the teachers and 
administrators, when the accountability results are likely to be seen in comparison to other 
districts, good intentions are impaired by the reality of the situations. This paper identified 
some specific policy issues related to implementation of the assessment and accountability 

system. , 

7f Multiple measures 

CDE requires that all districts use multiple measures for assessing student achievement 
and that student achievement should be reported based on the district adopted or developed 
performance standards. However, it is not clear as to how many measures are multiple and 
how many grade levels must have multiple measures. No criteria about the reliability and 
validity has been set by the state in using multiple measures. The directions at best are vague. 
The districts wanting to look good can use "inflated" class grades as one of the multiple 
measures. Particularly, class grades have little reliability and non-achievement criteria such as 
effort, attendance, and classroom participation are often factored into assigning class grades. 

Furthermore, there is a great deal of flexibility in weighting the measures. One can 
choose to give 70% of weight to class grades and only 30% to standardized test scores in 
judging student performance. This will paint a very different picture of student achievement 
from district to district in California. CDE needs to set clear policy guidelines in 
implementing the assessment and accountability system in order to ensure that there is an 
equity in the system forjudging student performance at or above grade level standards. 

2. Purpose of accountability 

CDE has not clarified the purpose of the accountability reports. The purpose has been 
specified as for identifying Coordinated Compliance Review (CCR) school visits, California 
School Recognition Program and Achieving School recognition program, and Title I 
identification of Program Improvement schools. It is not clear how the non-Title I schools 
accountability information will be used and the purpose of collecting the accountability report 
for these schools, besides using this information to identify schools as distinguished schools. 
One question that comes to mind is whether these results be used to compare districts and 
schools. Will these results be made public by releasing to the newspapers? If so, what kinds 
of information will be released to the newspapers? CDE needs to clarify the purpose of 
collecting accountability information and how these information will be used. 



3. 



CDE indecisions and district planning 






CDE informed the assessment and accountability procedure to the district on June 30, 
1997 in a memo from Ruth McKenna, Chief Deputy Superintendent. That was not soon 
enough for most districts to gather multiple measures of student achievement and report 
standards-based student performance by November 1, 1997. There should have been a 
reasonable amount of time for districts to plan and implement the assessment and 
accountability system and for staff development. Furthermore, the role of the new statewide 
norm-referenced standardized test, to be adopted by the State Board of Education on 
November 14, 1997, is not clear either. CDE should give firm, clear directions to districts 
about the role of the new test in the accountability system and more clear and precise 
directions for the use of multiple measures in order for district to collect valid and reliable 
student performance information. 

4. Support to districts and Program Improvement schools 

CDE has not been given enough capacity building to implement a new methodology of 
standards-based assessment and accountability system. It is not clear how and in what capacity 
the state will provide the technical support for nearly half of the Title I schools in the state that 
are likely to be identified as Program Improvement schools. The districts consider this 
approach of accountability very promising, but a great deal of effort at the local and state level 
is needed to make this system meaningful for all involved. 
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